The proximal policy optimization (PPO) algorithm stands as one of the most prosperous methods in the field of reinforcement learning (RL). Despite its success, the theoretical understanding of PPO remains deficient. Specifically, it is unclear whether PPO or its optimistic variants can effectively solve linear Markov decision processes (MDPs), which are arguably the simplest models in RL with function approximation.

The goal is to train a good instance classifier.

Low-rank matrix recovery is a fundamental problem in machine learning with numerous applications.

Consequently, standard online convex optimization algorithms do not directly apply.